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AUTOMATIC REGISTRATION OF INTRA-MODALITY MEDICAL 
VOLUME IMAGES USING AFFINE TRANSFORMATION 



The present invention relates to the diagnostic imaging arts. It finds 
particular sq)plication in conjunction with CT oncological studies of the lungs and will be 
described with particular reference thereto. However, it is to be appreciated that the 
present invention is applicable to a wide range of diagnostic imaging modalities and to the 
5 study of a variety of organs for a variety of reasons. 

When a patient is undergoing treatment for lung cancer, the lungs are 
periodically re-inspected, such as with a CT scanner. The oncologist compares the current 
images from the CT scanner with the images of the subject that were taken at an earlier 
time. Based , on this comparison, the oncologist can determine the rate of progression of 
10 the disease or whether it is in remission. 

Typically, the oncologist calls up both the current image data set and the 
prior image data set. The oncologist displays a transverse slice through the lungs of one of 
the image sets at the top of the video monitor and a transverse slice of the patient's lungs 
from the other data set at the bottom of the monitor. The oncologist manually steps 
IS through the two data sets independently and manually determines corresponding slices. 
Often, several thinner slices are fused into a thicker slice. 

One of the difGculties resides in the matching of corresponding slices. The 
matching is both time consuming and subjective. 

Moreover, the data from the two data sets may have been generated with 
20 different imaging parameters, with a different placement of the patient relative to the center 
of the scan circle, and the like. The two data sets may have slices of different thickness or 
skewed at different angles. The images may be shifted relative to the center of the scan 
circle. There may be differences in the field of view or scale of the images. The lungs 
may have been imaged at a different pulmonary phase. These and other such factors all 
25 contribute to the difficulty of manual image alignment and increase the prospect for 
subjectivity or human error. 
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Previously, technique have been developed that use fiducials to align two 
images. That is, fiducials or imageable markers are affixed to the patient closely adjacent 
the region of interest so that they are in a fixed relationship. These markers are readily 
aligned in the two images. 
S However, when the images are taken some duration apart, maybe months, 

maintaining the fiducials attached to the patient's exterior for that duration is inconvenient. 
Moreover, fiducials are used less commonly for internal organs in the torso because the 
patient surface moves in relation to most internal organs with respiration, body position, 
and the like. 

10 Images have also been aligned using anatomical markers. That is, 

characteristic portions of the body around the region of interest, such as unique locations 
on the skull or the vertebrae, have been used instead of externally applied fiducials. 
However, distinct anatomical markers are not readily available for lungs and other organs 
which can move relative to them. Consequently, most fiducial and anatomical marker 

15 alignment techniques have been used in rigid portions of the body, such as the head inside 
the rigid skull or along the vertebrae. 

The present application contemplates a new and improved automated 
registration technique which overcomes the above-referenced problems and others. 

20 

In accordance with one aspect of the present invention an apparatus for 
diagnostic imaging is disclosed. A first memory stores a first diagnostic image. A second 
memory stores a second diagnostic image. A means automatically registers the first and 
second diagnostic images firom the first and second image memories without operator 

25 assistance. A means for concurrently displays corresponding slices of the first and second 
registered diagnostic images. A means concurrently steps the displayed slice to 
corresponding regions of the first and second images. 

In accordance with another aspect of the present invention a method of 
diagnostic imaging is disclosed. A first and a second diagnostic images are stored. The 

30 first and second diagnostic images are automatically registered without operator assistance. 
Corresponding slices of the first and second registered diagnostic images are concurrently 
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displayed. The displayed slices are concurrently stepped to corresponding regions of the 
first and second images. 

One advantage of the present invention is that is quick, easy to use, and 

simple. 

5 Another advantage of the present invention is that it is completely 

automatic. 

Another advantage resides in its precision. 

Another advantage resides in the coordinated viewing and stepping through 
the images of both scans in a single action. 
10 Yet other advantages reside in the ease of follow-up examinations of the 

same patient. 

Still further advantages and benefits of the present invention will become 
apparent to those of ordinary skill in the art upon reading and understanding the following 
detailed description of the preferred embodiments. 

15 



The invention may take form in various components and arrangements of 
components, and in various steps and arrangements of steps. The drawings are only for 
purposes of illustrating the preferred embodiments and are not be construed as limiting the 
20 invention. 

FIGURE 1 is a diagrammatic illustration of an apparatus which 
automatically registers 3D images of a whole body part; 

FIGURE 2 is an expanded diagranunatic illustration of a part an apparatus 
of FIGURE 1 which automatically registers 3D images of a whole body part; 
2$ FIGURE 3 is a diagrammatic illustration of the affine transform means of 

FIGURE 1 and associated parts of the apparatus; 

FIGURE 4 is an expanded diagrammatic illustration of a part the apparatus 
of FIGURE 1 which automatically registers 3D images of a whole body part and displays 
them on the monitor. 

30 
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With reference to FIGURE 1, a patient is positioned in a diagnostic imager 
10, such as a CT scanner, for a follow-up examination. The generated data is reconstructed 
by a reconstruction processor 12 and stored in a 3D, volumetric image memory 14. 
Various image enhancement operations as are known in the art are also performed. 
S Image data from the hospital archive or from another storage medium 16 of 

the same region of the same patient is retrieved and stored in an archived 3D volumetric 
image memory 18. Of course, both the current and archive 3D image memories may be 
parts of a common storage medium. A registering means 20 retrieves the current and 
archived images and automatically register the two images for a concurrent display on a 
10 monitor 22. 

With continuing reference to FIGURE 1 and ftirther reference to FIGURE 
2, a surface image generating means 24 generates surface images from the current archived 
images. A segmenting means 26 extracts the important body parts for the registration, then 
a feature extraction means 28 extracts common features for registration. Looking 

15 specifically to lung images, the segmentation includes a thresholding means 30 thresholds 
the images. More specifically to the lung embodiment, the thresholding means 30 sets the 
gray scale value of air in the lungs to one extreme value, e.g., black or white, and all tissue 
in the image to the other. Specifically to lung images, the feature extraction means 28 
includes a surface voxel extracting means 32 which extracts the single layer of voxels 

20 which lies along the interface between the black and white regions. Preferably, the voxels 
which represent the tissue surface rather than the air surface are selected. A feature 
exclusion means 34 removes part of the features of the 3D image that is defective or 
artifacted, if any. Looking specifically to lung images, portions of the lung closely 
adjacent to the heart often suffer motion artifacts. To simplify the remaining operations, 

25 the feature exclusion means 32 optionally removes a common artifacted region from both 
image representations. The voxels are stored in feature or surface image memories or 
memory sections 36, 36'. 

A scaling means 40 scales the two surface images in accordance with 
known differences in scanning parameters. For example, the images are adjusted for 

30 differences in the slice thickness of the scans, for the field of view, for magnification 
differences, and the like. A normalizing means 42 normalizes the two surface images. 
More specifically to the preferred embodiment, the extreme points along the z-axis of each 
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image are labeled -1 in one direction and +1 in the other direction. Points in between are 
labeled proportionately. The same scaling is carried out with respect to the x and y-axes. 
Different normalization methods are also available, for example, computing the mean or 
median and standard deviation of the data and setting the central point to the mean or 
5 median and the length of one is set to the standard deviation from the mean or median. Of 
course, other coordinate systems are contemplated. The two scaled and normalized three- 
dimensional surface images are stored in scaled and normalized volume image memories 
or memory sections 44, 44'. 

With continuing reference to FIGURES 1 and 2 and further reference to 

10 FIGURE 3, an affine transform means 50 determines the twelve values of the afiine 
transform that defines the registration error between the current and archived scaled and 
normalized 3D images. Specifically, the affine transform means 50 determines nine 
rotational components about three orthogonal axes and three translational components 
along the three axes. Optionally, a scaling parameter can also be determined. 

15 The affine transform means 50 includes a selective or non-uniform random 

feature or point reducing means 52. That is, with nomial CT resolution, the two surface 
images are still hundreds of thousands of voxels. To reduce the computational load, the 
selective (based on prior knowledge) point reducing means 52 or a non-unifomi random 
point reducing means 52 preferably selectively (based on prior knowledge) randomly 

20 selects a fraction of the points, e.g., 1%, from both images. Because the diagnosing 
oncologist typically is most concerned with registration along the z- or longitudinal axis, 
the non*uniformity with which the points are selected is altered to favor points, which 
influence the z-registration. In the preferred embodiment, a higher percentage of points are 
selected along the top and bottom axes of the lungs. A matching means 54 matches the 

25 non-uniform randomly selected points from the one surface image with a corresponding 
point in the other surface image. In the preferred embodiment, a K-D tree matching 
technique or processor 56 is utilized. In each iteration, a non-uniformly random sample of 
the points from the current image, to be matched to the corresponding point in the other 
surface image, is taken to be matched to the other set of points from the previous image. 

30 For each point in the sample, a matching point is found using the K-D tree. For each 
matching pair, the coordinates or location of each voxel of the pair and the surface normal 
are determined. In the matching operation, if more than one candidate point has an equally 
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good spatial match to the reference point, the point or voxel with the closest normal is 
selected to generate a set of points and normals which are stored in a point and normal 
memory means 58. 

A bad pair removal means 60 removes matched pairs of points that fail to 
S meet preselected criteria. In the preferred embodiment, these criteria include a physical 
separation distance greater than a preselected minimum, badly misaligned normals, and 
points near to the cut surface, if any. An error estimating means 62 estimates the error 
between the points. The error estimating means 62 calculates a weighted distance using 
the differences in x, y, z, and the normals. Each coordinate is assigned with different 

10 weight so that the differences in specific parts will be favored over other parts. Looking 
specifically to lung images, the differences in the z-coordinate have higher weight than the 
X and y-coordinates, and higher than the normal differences. 

With continuing reference to FIGURE 3, an error minimizing or affine 
registration means 70 finds an affine transform that minimizes the error between the pairs 

15 of points. In the preferred embodiment, the error minimizing means 70 uses a 
Levenberg-Marquardt minimization technique or processor 72. Using the weighted error 
estimate and the divergence of the weighted error estimate, the error minimizing means 
finds 70 a set of affine transformation parameters that gives the minimum of error for the 
set of sample pairs. Twelve affine transform coordinates and optionally the scaling 

20 correction are loaded into a correction transform memory 80. 

With continuing reference to FIGURE 3, a transform processor 82 
transforms one of the current and archived scaled and normalized surface images from 
memories 44, 44' in accordance with the determined correction transform. The affine 
transform means 50 repeats the registration technique re-registering the volume images, 

25 with one of them subject to the transform, which was predicted, to transform them into 
registration. This process generates the twelve affine transform coordinates (and 
optionally scaling) of a correction transform to the first correction transform. An end 
criteria determining means 84 determines whether the two images have been aligned 
within acceptable tolerances by monitoring the correction transforms. In the preferred 

30 embodiment, the end criteria determining means 84 determines whether the global 
translation is less than a preselected threshold, whether the change in the correction 
transform is below a preselected threshold, or whether a preselected number of iterations 
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have been performed. If any of these criteria are met, a correction transform combining 
means 86 combines the first correction transform and each of the subsequent correction 
transforms to provide a single registration transform which is stored in a registration 
transfoim memory or buffer 88. 
5 With reference to FIGURE 1 again and further reference to FIGURE 4, a 

transform operator 90 operates on one of the current and 3D archive images, preferably the 
archived image to register it with the current 3D volume image. The transformed archived 
unage is stored in a transformed 3D image volume memory 92. 

An operator, through a keyboard or other input device 100 controls a 

10 stepping means 102 which causes a display processor 104 to withdraw and display 
corresponding slices from the current 3D image memory 14 and the transformed 3D image 
memory 92 on a monitor 22. In the preferred embodiment, the transverse slice from the 
current image memory 14 is displayed on the top half of the screen and the corresponding 
slice of the transformed memory 92 is displayed on the bottom half of the screen. 

15 Preferably, corresponding orthogonal views are displayed adjacent the larger longitudinal 
slice. The operator input device 100 also controls a slice fusing means 106 that works with 
the display processor 104 to sum a preselected number of adjacent slices in each of the two 
images. The operator input means 100 is also connected with a single image slice stepping 
means 108, which can control the display processor 104 to step through slices of only one 

20 of the two 3D images independently of the other. 

Typically, 3-5 slices are fused together for display on monitor 22. A global 
registration algorithm which registers the two three-dimensional volume unages within ±2 
slices is considered accurate registration due to the fusing of the number of adjacent 
images. 

25 Although described with particular reference to CT scanner imaging, it is to 

be appreciated that this technique is also applicable to magnetic resonance images, PET 
images, SPECT images, and other three-dimensional diagnostic images. Moreover, the 
images being registered may be from mixed modalities. For example, a CT image can be 
registered using this technique with a PET image. When mixing modalities, care should be 

30 taken to assure that the features are defined in both imaging modalities or appropriate 
adjustment made. Further, although described with particular reference to the lungs, it is to 
be appreciated that this technique is also applicable to studies of other organs such as the 
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colon, the liver, and other non-rigid organs. Moreover, this technique is also applicable to 
rigid portions of the body such as the head. 

The invention has been described with reference to the preferred 
embodiments. Modifications and alterations will occur to others upon a reading and 
S understanding of the preceding detailed description. It is intended that the invention be 
construed as including all such modifications and alterations insofar as they come within 
the scope of the appended claims or the equivalents thereof. 
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